{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "55050206-2e36-4c81-855c-3a2b2cce2b71",
   "metadata": {},
   "source": [
    "# Adapting metrics to target language"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88a4b371-5d9e-4792-a2a8-3298858e86f6",
   "metadata": {},
   "source": [
    "While using ragas to evaluate LLM application workflows, you may have applications to be evaluated that are in languages other than english. In this case, it is best to adapt your LLM powered evaluation metrics to the target language. One obivous way to do this is to manually change the instruction and demonstration, but this can be time consuming. Ragas here offers automatic language adaptation where you can automatically adapt any metrics to target language by using LLM itself. This notebook demonstrates this with simple example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9ee174e-3282-4188-80fc-05e2f94c0cc3",
   "metadata": {},
   "source": [
    "For the sake of this example, let's choose and metric and inspect the default prompts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "483b2b0d-46da-48b5-8e3f-329dc84205b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.metrics import SimpleCriteriaScoreWithReference\n",
    "\n",
    "scorer = SimpleCriteriaScoreWithReference(\n",
    "    name=\"course_grained_score\", definition=\"Score 0 to 5 by similarity\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "df672d64-273e-4f74-8d5a-0a76d777bf22",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'multi_turn_prompt': <ragas.metrics._simple_criteria.MultiTurnSimpleCriteriaWithReferencePrompt at 0x7fcf409c3880>,\n",
       " 'single_turn_prompt': <ragas.metrics._simple_criteria.SingleTurnSimpleCriteriaWithReferencePrompt at 0x7fcf409c3a00>}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scorer.get_prompts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efb649fb-900c-4edb-a92e-65a551180953",
   "metadata": {},
   "source": [
    "As you can see, the instruction and demonstration are both in english. Setting up LLM to be used for this conversion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "84bdba84-f046-4964-b141-02b7aa59b1c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.llms import llm_factory\n",
    "\n",
    "llm = llm_factory()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "358366cd-fa4e-44c4-9f75-c441356e1554",
   "metadata": {},
   "source": [
    "Now let's adapt it to 'hindi' as the target language using `adapt` method.\n",
    "Language adaptation in Ragas works by translating few shot examples given along with the prompts to the target language. Instructions remains in english. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "98bb3f1a-8b63-480a-9aaf-474e2cf82df3",
   "metadata": {},
   "outputs": [],
   "source": [
    "adapted_prompts = await scorer.adapt_prompts(language=\"hindi\", llm=llm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a42236a-d084-4bab-9728-e65e680ef10e",
   "metadata": {},
   "source": [
    "Inspect the adapted prompts and make corrections if needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5cc2f37b-da93-4184-b7fb-9fa870943f43",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'multi_turn_prompt': <ragas.metrics._simple_criteria.MultiTurnSimpleCriteriaWithReferencePrompt at 0x7fcf42bc40a0>,\n",
       " 'single_turn_prompt': <ragas.metrics._simple_criteria.SingleTurnSimpleCriteriaWithReferencePrompt at 0x7fcf722de890>}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adapted_prompts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dd789f1-f4cb-4e61-9a99-4343e25a4bc3",
   "metadata": {},
   "source": [
    "set the prompts to new adapted prompts using `set_prompts` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "afb18db0-fe0e-46f4-9fc2-a80ec3141359",
   "metadata": {},
   "outputs": [],
   "source": [
    "scorer.set_prompts(**adapted_prompts)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "958f8ab5-89a1-46d8-b836-cc19712b51c7",
   "metadata": {},
   "source": [
    "Evaluate using adapted metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "85c01c6e-9f07-4a4e-9c3a-50c302c51fd4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from ragas.dataset_schema import SingleTurnSample\n",
    "\n",
    "sample = SingleTurnSample(\n",
    "    user_input=\"एफिल टॉवर कहाँ स्थित है?\",\n",
    "    response=\"एफिल टॉवर पेरिस में स्थित है।\",\n",
    "    reference=\"एफिल टॉवर मिस्र में स्थित है\",\n",
    ")\n",
    "\n",
    "scorer.llm = llm\n",
    "await scorer.single_turn_ascore(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f00812e9",
   "metadata": {},
   "source": [
    "Trace of reasoning and score\n",
    "\n",
    "`{\n",
    "    \"reason\": \"प्रतिक्रिया और संदर्भ के उत्तर में स्थान के संदर्भ में महत्वपूर्ण भिन्नता है।\",\n",
    "    \"score\": 0\n",
    "}`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6ffbe6c",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ragas",
   "language": "python",
   "name": "ragas"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
